1. Field of the Invention
The present disclosure relates generally to packet network devices such as switches and routers, and more particularly to methods for stacking multiple such devices to form a larger logical device.
2. Description of Related Art
Packet network devices direct data packets traveling across a network between data sources and destinations. Packet network devices can perform “routing” or “switching” depending on the header information and networking techniques used to direct the data packets. A single packet network device may be configured to perform both switching and routing. Such devices are referred to herein as a “packet switch” with the understanding that this term encompasses a wide variety of packet forwarding capabilities.
FIG. 1 shows a high-level block diagram for an exemplary modular packet switch 100. The switch comprises a chassis (not shown) with an electrical backplane 110, to which circuit cards mate in order to communicate with other cards in the chassis. In FIG. 1, the switch accommodates n line cards, LC1-LCn, m switch fabric cards, SFC1-SFCm, and two route processing managers RPM0, RPM1.
Each line card LCx (where x represents any of the line card slot positions 1 to n) receives ingress data traffic from and transmits egress data traffic to peer devices through its external ports to a port interface, PIx. Port interface PIx transmits and receives the physical layer signaling required by the external ports, performs framing, and communicates packet streams for each external port with a packet processor PPx. The ports can be configured for different electrical or optical media via the use of different line card types, different port interface modules, and/or different pluggable optics modules.
For most ingress packet traffic on each line card LCx, a line card packet processor PPx processes the packet, determines one or more switch egress ports for the packet, and queues the packet for transmission through the switch fabric when possible. For most egress packet traffic on each line card LCx, the line card queues the packets arriving from the switch fabric, and selects packets from the queues and serves them fairly to the egress ports. To accomplish these functions, PPx interfaces with a scheduler interface SCHx, a content-addressable memory CAMx, and a line card processor LCPx. PPx notifies scheduler interface SCHx as to the destination line cards for which it has packets queued, and receives switch fabric schedule information back from SCHx. PPx uses the scheduling information to place packets from one or more queues on the switch fabric backplane connections during each scheduling epoch. The switch fabric connections are implemented using unidirectional differential serial bus pairs, with each line card packet processor having at least one ingress pair for each switch fabric card and one egress pair for each switch fabric card. An intermediate serializer/deserializer (“serdes,” not shown) may be used between PPx and the backplane differential serial pairs.
Memory CAMx stores lookup tables that PPx accesses to determine what operations to perform on each packet, as well as the next hop destination for each packet. PPx generally constructs one or more lookup keys from the headers and receive port information, etc., and performs one or more lookup operations on CAMx. CAMx returns pointers into instruction memory and next hop memory (not shown), as well as any parameters needed to process the packets.
Line card processor LCPx is a general purpose processor that handles control plane operations for the line card. Control plane operations include programming CAMx and lookup memory according to instructions from the master RPM, programming registers on PPx that tailor the line card behavior, receiving control plane packets (packets addressed to switch 100, e.g., for various routing/switching protocols) from PPx, and transmitting control plane packets (packets generated by switch 100 for communication to a peer device) to PPx for forwarding out an external port. LCPx may implement some control plane functionality for some protocols handled by switch 100.
LCPx also connects to the electrical backplane through a card management FPGA (Field Programmable Gate Array) FPGAx and an Ethernet switch ESWx. The card management FPGA connects through serial management bus interfaces on electrical backplane 110 to master card management FPGAs on the RPMs. The RPMs use the serial management bus to boot line cards, monitor the health of the line card and its environmental parameters, manage power for the line card and its components, and perform basic hardware configuration for the line card. Various registers on FPGAx are readable by line card processor LCPx, allowing LCPx to perform some of its configuration tasks using values stored in the registers by the RPMs.
Ethernet switch ESWx connects to two backplane Ethernet point-to-point links, one linking ESWx to each RPM. LCPx transmits control plane traffic, as well as line card statistics and monitored data plane traffic, to the master RPM using packets transmitted to ESWx. Likewise, the master RPM transmits FIB (Forwarding Information Base) updates and boot images to LCPx for consumption, and control plane packets destined for external peers to LCPx for forwarding to PP1.
Switch fabric card SFC1 is exemplary of the switch fabric cards, which are generally identical in a system. Switch fabric card SFC1 comprises a switch fabric device, SF1, a switch fabric scheduler interface SFSCH1, and a card management FPGA1. Although FPGA1 can be the same type of hardware device as FPGA1 on line card LC1 and have the same RPM interface programming, its other programming is generally specific to the requirements of a switch fabric card. The scheduler interface SFSCH1 does not provide schedule information to each RPM, it merely receives the epoch schedules used to program the switch fabric for each epoch. The switch fabric cards work in parallel according to a common epoch schedule. Each epoch, every switch fabric card has a switch fabric SFy that is programmed the same way through the switch fabric scheduling interface SFSCHy.
The switch fabric SF1 provides serdes interfaces for each line card and a parallel crossbar switch that can switch any of the inputs to any number of the outputs. In one embodiment, the line cards slice up each packet sent to the switch fabric into lanes, sending one byte of the packet to SFC1, the next byte to SFC2, etc., in round-robin fashion on each lane. A receiving line card aligns the lanes incoming from the switch fabric cards and reassembles the packet data. This allows the switch to tolerate malfunctioning switch fabric cards by simply not sending data on a malfunctioning lane.
The route processing managers (RPMs) control all aspects of the overall operation of the chassis. RPM1 and RPM2 are identical, and the switch can run with only one RPM. When two RPMs are present, one is elected as the master, and the other remains on standby. The standby RPM monitors the health of the master, and takes over as master should the first fail. Each RPM RPMx comprises three processors: a control processor CPx, which controls the overall operation of the switch; and two route processors RPx.0, RPx.1, which run different routing/switching protocols, communicate with external peers, and program the line cards to perform correct routing and switching.
Each RPM RPMx also has three bus interfaces to the electrical backplane. A master Ethernet switch MESWx connects through backplane Ethernet links to each line card Ethernet switch, ESWy, and to the control processor and each route processor. A master scheduler MSCHx connects through the backplane scheduling bus to each line card and switch fabric card—the master scheduler determines from the line card requests a switch fabric schedule for each epoch, and communicates this schedule to the line cards and switch fabric cards. A master FPGA MFPGAx connects through backplane serial management bus connections to each other card in the chassis. The master FPGA monitors the health of each other card through heartbeat message exchanges and environmental data collections, and provides an interface for CPx to control the hardware on each card. In some hardware implementations, each RPM also has serdes interfaces to lanes on the switch fabric cards, and can receive and send packets to the line cards over the switch fabric.
As alluded to above, communications between the chassis modules uses four separate bus systems on electrical backplane 110. A packet data bus system connects the line cards and the switch fabric cards, allowing high-speed transfer of data traffic through the switch. Due to the high data rates of this bus and the fact that separate trace groups connect each of n line card slots to each of m switch fabric card slots, this is the largest bus system on the electrical backplane 110. A control plane packet data bus system allows the master routing/switching protocols running on the RPMs to communicate with each other, with the line cards, and with peer devices through the line card external ports. This packet data bus system may be, e.g., a point-to-point Ethernet link between each line card and each RPM card. A scheduling bus system allows the master RPM to coordinate and synchronize the internal transmission of packet data between the line cards and the switch fabric cards. This bus system is separated to preserve critical timing for the exchange of scheduling information between each line card and switch fabric card and each RPM. And a chassis management bus system allows the RPMs to monitor, configure, and manage the hardware on each line and switch fabric card. This bus system may use industry standard relatively low-speed serial management bus protocols, such as System Management Bus (SMBus), across point-to-point links between each line card or switch fabric card FPGA and each master FPGA.